Improving the State Space Organization of Untrained Recurrent Networks
نویسندگان
چکیده
Recurrent neural networks are frequently used in cognitive science community for modeling linguistic structures. More or less intensive training process is usually performed but several works showed that untrained recurrent networks initialized with small weights can be also successfully used for this type of tasks. In this work we demonstrate that the state space organization of untrained recurrent neural network can be significantly improved by choosing appropriate input representations. We experimentally support this notion on several linguistic time series.
منابع مشابه
Approaches Based on Markovian Architectural Bias in Recurrent Neural Networks
Recent studies show that state-space dynamics of randomly initialized recurrent neural network (RNN) has interesting and potentially useful properties even without training. More precisely, when initializing RNN with small weights, recurrent unit activities reflect history of inputs presented to the network according to the Markovian scheme. This property of RNN is called Markovian architectura...
متن کاملText Correction Using Approaches Based on Markovian Architectural Bias
Several authors have reported interesting results obtained by using untrained randomly initialized recurrent part of an recurrent neural network (RNN). Instead of long, difficult and often unnecessary adaptation process, dynamics based on fixed point attractors can be rich enough for further exploitation for some tasks. The principle explaining untrained RNN state space structure is called Mark...
متن کاملRecurrent Kernel Machines: Computing with Infinite Echo State Networks
Echo state networks (ESNs) are large, random recurrent neural networks with a single trained linear readout layer. Despite the untrained nature of the recurrent weights, they are capable of performing universal computations on temporal input data, which makes them interesting for both theoretical research and practical applications. The key to their success lies in the fact that the network com...
متن کاملRole of Local Economy in Excess Flow and Reconstruction of Regional Network System Case: Kerman Province
Introduction Changes in spatial patterns and its structural and functional dynamics which is apparently obvious within physical links and relationships of cities and rural and urban network is fundamentally one of the outcomes of forces that contribute to the controlling process of actual mechanism of capital absorption, its flow, work forces, and population. Such forces play a significant r...
متن کاملThe peculiarities of art space worldview in I. S. Shmelev’s novel “The Lord’s Summer”
The article looks into the global literary perspective presented in I. S. Shmelev’s novel “The Lord’s Summer” through the author’s special organization of the time scene, the characters’ language or speech system and genre specific features as well. In particular, it is noted that, through spatial reference points the author's system of values is expres...
متن کامل